_ 10/501253 

'^^pf ^ i> ^>T04 Rsc^d PCT/FTO 0 8 JUL 2004 

[10191/3759] 



METHOD AND DEVICE FOR RECOGNIZING 
VISUAL OBSTRUCTIONS IN IMAGE SENSOR SYSTEMS 



Background Information 

The present invention relates to a method and to a device for 
recognizing visual obstructions in image sensor systems. 

5 

In the future, in particular in conjunction with environmental 
sensing of motor vehicles, image sensors such as video 
cameras, for example, will be used whose image is evaluated by 
downstream functions (driver assistance functions, for 
10 example) . It is therefore of particular importance to 

recognize visual obstructions which degrade the image detected 
by the image sensor, to inform the downstream functions or 
systems of the presence of such visual obstructions, and to 
initiate countermeasures if necessary. 

15 

Advantages of the Invention 

The detection of the blurriness of the recorded image results 
in a suitable procedure which assists in recognizing visual 

20 obstructions. It is particularly advantageous that downstream 
systems or functions, which function properly only when the 
view is adequate, are informed of the presence of the visual 
obstruction and are thus able to initiate suitable 
countermeasures. When a visual obstruction is recognized, 

25 depending on the design of the downstream function or system 
the recorded image is advantageously not evaluated or 
corrected, or other suitable measures, such as switching on 
the windshield wiper system or windshield heating, for 
example, are performed. 

30 
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It is particularly advantageous that visual obstructions are 
recognized based on the image signal itself, so that no 
additional sensors are necessary. 

5 The procedure is suitable in particular for recognizing a 
visual obstruction on the windshield of the motor vehicle. 

The image sensor advantageously takes over additional 
functions, such as the function of a rain sensor, for example, 
10 so that this rain sensor may be dispensed with in the 

furnishing of the vehicle with such image sensor-based driver 
assistance functions, for example lane monitoring systems. 

The procedure described below is used in a particularly 
15 advantageous manner in video sensor systems in motor vehicles 
which are not focused on the vehicle windshield, but rather on 
the external region. The procedure described below may thus be 
used to particular advantage in conjunction with the sensing 
of the environment around a motor vehicle. In this manner the 
2 0 image sensor may be used for multiple applications (for 
example, rain sensor and object detection, etc.). 

It is also advantageous that the image sensor is able to check 
its functionality using its own output signal, i.e., to 
25 determine whether the instantaneous view conditions are 

sufficient for the next function that is to be performed. When 
the image sensor is not functioning properly, this is 
advantageously signaled to the driver or the downstream 
system, and/or countermeasures are taken. 

30 

It has been shown in a particularly advantageous manner that 
the detection of visual obstructions is performed by measuring 
the blurriness of the represented image . A reliable method is 
thus provided for determining visual obstructions, in 
35 particular objects on a vehicle windshield, which appear 
blurry in the image. It is particularly advantageous that 
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transpajrent objects such as raindrops, or semi transparent 
objects such as ice or dust, for example, are recognized, and 
in one particular embodiment even distinguished from one 
another, by measuring the blurriness of the image. 

Thus, it is also advantageously possible to obtain information 
about the type of visual obstructions. In this manner a 
response specific to the particular type of visual obstruction 
may be made, for example by automatically switching on the 
windshield washer system when the windshield is soiled by 
particles . 

Further advantages arise from the following description of 
exemplary embodiments and the dependent claims- 



The present invention is explained in greater detail below 
with reference to the exemplary embodiments illustrated in the 
drawing . 



Drawing 



Figure 1 



shows an overview illustration of an image sensor 
system in which the procedure described below for 
determining visual obstructions is implemented; 



and 



Figures 2 
through 4 



show diagrams with reference to which three 
different embodiments for determining the 
blurriness in the image and the recognition of 
visual obstructions derived therefrom are 



described. 



Description of the Exemplary Embodiments 



The image sensor system illustrated in Figure 1 is composed 
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essenti-ally pf an optical sensor array which is connected to a 
data processing unit 7, for example a microprocessor. The 
instantaneously visible scene is mapped onto the sensor array 
by a lens system 2 . The image sensor is mounted behind a 
5 windshield 1 and is focused on the external region behind the 
windshield. The windshield is, for example, the window glass 
of a vehicle, in particular the windshield of the vehicle. The 
focusing on the vehicle's external region allows the image 
sensor system to also be used for other image processing 

10 operations such as lane detection or obstruction detection, 
for example. Evaluation unit 7 is composed of a module 4 for 
controlling lighting of the sensor array, a measuring unit 5 
for estimating blurriness, and a decision module 6 which makes 
decisions regarding the presence and, optionally, the type, of 

15 a visual obstruction. In one preferred embodiment all of these 
modules are part of a microprocessor, the modules being 
implemented as software programs. In other embodiments, the 
modules are individual components (processors) , likewise 
implemented as software, where estimations of blurriness are 

2 0 made, for example, in one component and decisions are made in 

another component. The decision made by decision module 6 is 
then relayed as an information and/or activation signal to 
downstream functions and/or systems. Application possibilities 
are described in greater detail below. 

25 

The automatic detection of visual obstructions on the 
windshields of a vehicle therefore plays an increasingly 
important role due to the fact that, when image sensors are 
used in the vehicle, systems which evaluate the images from 

3 0 the sensor function properly only when the view is adequate. 

If there is information about the visual obstruction and, in 
the ideal case, about the type of visual obstruction, such a 
system is able to display to the driver its instantaneous lack 
of functionality and/or to take countermeasures, such as 
35 switching on the wiper system, windshield heating, windshield 
washer system, for example. 
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When visual obstructions are determined based solely on the 
image sensor signal, the problem arises that visual 
obstructions on a vehicle windshield are made noticeable in 
the image only indirectly, since for use in driver assistance 
5 systems the image sensor is focused on the vehicle's external 
region. Objects on the windshield, for example raindrops or 
dust, are therefore blurrily imaged. These visual obstructions 
are thus made noticeable by a characteristic distribution of 
the blurriness in the image signal. 

10 

The detection of visual obstructions is based on measuring the 
blurriness of the image which is recorded by an image sensor. 
This blurriness is made noticeable by a lack of sharpness of 
the imaged contours of the instantaneously visible scene. 

15 Objects present on the windshield are blurrily imaged due to 
the focusing of the image sensor on the external region. 
Transparent objects, such as raindrops, for example, result in 
localized defocusing of the image, and semitransparent objects 
such as ice or dust scatter incident light beams. Both effects 

20 result in an increase in the blurriness of the sensor signal. 

The evaluation unit which is associated with the image sensor, 
an optical sensor array of, for example, a CCD or CMOS camera, 
detects the distribution of blurriness of the instantaneously 
imaged scene. Based on the detected distribution of 

25 blurriness, a conclusion is drawn as to the presence of a 
visual obstruction. One of the methods for digital image 
processing described below is used for deteannining the 
blurriness . 

3 0 In one preferred embodiment, the image sensor is a sensor 

array which assists in obtaining information also on the type 
of visual obstruction. This fundamentally distinguishes such a 
sensor array from a conventional rain sensor which operates by 
the reflection principle. The use of a sensor array allows a 

35 specific response to be initiated to the determined visual 
obstruction, for example, by additionally switching on a 
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windshijeld washer system for soiling caused by particles. One 
preferred procedure for recognizing the type of visual 
obstruction is likewise described in detail below. 

A first possibility for measuring blurriness is illustrated 
with reference to the diagrams in Figure 2. Blurriness is 
measured here using the contrast spectrum. The underlying 
principle of this method is known from the field of digital 
image processing. To obtain the contact spectrum, first a 
multiscale analysis is carried out in which the video image is 
decomposed into multiple images of decreasing resolution by 
the repeated use of a smoothing operation and subsequent 
subsampling. A global measure of contrast, for example the 
standard deviation of the intensity values in the image, is 
calculated in each resolution stage. The measure of contrast 
plotted as a function of the resolution forms the contrast 
spectrum of the image . Examples of such contrast spectra are 
shown in Figures 2a and 2b. In these figures, the particular 
measure of contrast K of the image is plotted as a function of 
resolution A. Sharp and blurry images differ in their contrast 
spectra by the fact that the drop in contrast as a function of 
increasing resolution is steeper in blurry images than in 
sharp images. This is because fine details are degraded more 
strongly by visual obstructions than are coarse image 
features. Thus, the drop in the contrast spectrum is a measure 
of the blurriness of the image. Situations are shown as 
examples in Figures 2a and 2b. Figure 2a illustrates a 
contrast spectrum as a function of the image resolution for an 
image with slight blurriness, while Figure 2b shows the 
situation for high image blurriness (blurry image) . 

In one implementation of the procedure for determining the 
blurriness of the image, the contrast spectrum is recorded by 
multiscale analysis within a computing program, and a variable 
is determined, for example the average increase in the measure 
of contrast as a function of the resolution, which 
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characterizes the variation of the measure of contrast as a 
function of the resolution. The blurriness of the image is 
determined by comparing this variable to at least one limiting 
value. For recognizing a visual obstruction, in particular 
5 with regard to the response measures planned in the downstream 
systems, it is sufficient to determine that the image is 
blurry; thus, the determined variable is compared to a 
limiting value, and if this limiting value is exceeded, 
blurriness may be assumed, or the variable itself is relayed 

10 as a measure of blurriness. In other applications, determining 
a measure of the blurriness is desired. In these cases, 
limiting values form ranges for the variable which 
characterizes the contrast spectrum and to which blurriness 
values are assigned. A value for the magnitude of the 

15 blurriness is determined when the value characterizing the 
contrast spectrum is in a specified range of values. 

In addition to measuring the blurriness by using the contrast 
spectrum, there are a number of other methods from the field 

2 0 of digital image processing that may be used to detect 

blurriness. Examples of such alternative methods for measuring 
the image blurriness are measures for analyzing the Fourier 
spectrum or the autocorrelation of the analyzed image. In the 
Fourier spectrum a blurry image is characterized by the fact 
25 that the amplitudes of the high spatial frequencies, which 

represent the finer image details, are strongly attenuated in 
comparison to a sharp image . 

Examples of such a situation are shown in Figures 3a and 3b. 

3 0 The curve of the Fourier spectrum is plotted in both figures, 

the amplitude of the Fourier transformation function being 
represented by the spatial frequency. Figure 3a shows a 
slightly blurry image, while Figure 3b shows a highly blurry 
image. It is obvious that the amplitudes at the higher spatial 
35 frequency decrease greatly for a blurry image. For evaluation, 
in one preferred embodiment it is therefore provided that the 
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amplitude and the corresponding spatial frequency are used to 
recognize values that fall below a threshold value. If this 
spatial frequency is below a predetermined value, it is 
assumed that the image is blurry. In another embodiment, the 
5 slope of the curve could also be used here to determine 

blurriness by comparing it to at least one limiting value. For 
the evaluation of the Fourier spectrum as well, it is possible 
to determine a value for the magnitude of blurriness by 
specifying multiple limiting values or by specifying value 
10 ranges, and relaying this value to downstream systems. 

A third possibility for measuring blurriness is the 
autocorrelation function of the image. This autocorrelation 
function decreases less steeply for a blurry image than for a 
15 sharp image as a function of distance. This is because only 
large structures remain in a blurry image. Figure 4 
illustrates this relationship, the particular magnitude of the 
autocorrelation function being plotted as a function of the 
pixel separation. Figure 4a shows the autocorrelation function 

2 0 for a slightly blurry image, and Figure 4b shows the 

autocorrelation function for a highly blurry image. The 
figures show that as the pixel spacing for an image having a 
low degree of blurriness (sharp image) increases, the 
autocorrelation function rapidly decreases, whereas for a 
25 blurry image the decrease in the autocorrelation function is 

less. As indicated above, for the evaluation it is possible to 
use at least one limiting value or multiple limiting values 
being exceeded or not being attained, or, in another 
embodiment, to use the calculated average increase to 

3 0 determine the blurriness and/or to deteinnine a value for the 

magnitude of the blurriness. 

A further alternative for measuring blurriness is provided by 
an indirect classification formulation in which a learning 
35 machine, a neural network or a polynomial classifier, for 

example, is trained to distinguish between blurry and sharp 
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images by presenting a large number of images as examples. 

As mentioned above, a visual obstruction is assumed when a 
high degree of blurriness is recognized, or the value of the 
5 blurriness is greater than a predetermined value. In addition, 
a comparison to reference distributions for contrasts, Fourier 
components, or autocorrelations may differentiate various 
types of visual obstructions based on their similarity to 
specific reference distributions. In this manner it is 

10 possible to distinguish moisture on the windshield from ice or 
dust on the windshield, and, depending on the situation, to 
initiate different countermeasures, for example, activation of 
the windshield wiper, switching on a windshield heating system 
or a windshield washer system, etc. Here as well, the above- 

15 referenced classification formulation may be provided by 
learning machines . 

If the video sensor having visual obstruction detection is 
also operated as a rain sensor, further additions are useful. 

20 If moisture on the windshield is recognized based on the 

determined blurriness and the contrast distribution, or the 
Fourier spectrum or autocorrelation, the windshield wiper is, 
as previously, initially actuated only at the explicit request 
of the driver, to prevent erroneous start-up when an error is 

25 detected. Using the video images which are recorded 

immediately after the wiping operation, it is possible to 
obtain the above-mentioned reference distributions, using 
which a decision is made to initiate the next wiping 
operation. Thus, the wiping response may be adapted on a 

30 situation-specific basis. For situations in which the 

surrounding scene contains too little contrast, it may be 
practical to temporarily switch on a windshield light. Visual 
obstructions are then detectable from the scattering of the 
light. 
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